How many rectangles are there whose four vertices are points on this grid? [asy]
size(50);
dot((0,0));
dot((5,0));
dot((10,0));
dot((0,5));
dot((0,10));
dot((5,5));
dot((5,10));
dot((10,5));
dot((10,10));
[/asy]
Explanation: We consider the different cases:

$\bullet$ $1$ by $1$ square: There are $4$ of those (formed by joining adjacent points).

$\bullet$ $2$ by $2$ square: There is $1$ of those (formed by joining the $4$ corner points).

$\bullet$ $\sqrt{2}$ by $\sqrt{2}$ square: There is $1$ of those (formed by joining the middle points on the $4$ edges, in other words, the diagonals of the $1$ by $1$ squares).

$\bullet$ $1$ by $2$ rectangle: There are $4$ of those.

So that's a total of $4+1+1+4 = \boxed{10}.$